Tag

#SWE-Bench Pro

2 articles

OpenAI finds roughly 30 percent of popular AI coding test is broken

OpenAI has found that around 30% of tasks in the popular SWE-Bench Pro benchmark are broken, leading the company to withdraw its endorsement of the test.

Jul 97

Separating signal from noise in coding evaluations

OpenAI's analysis reveals significant methodological flaws in SWE-Bench Pro, a popular coding benchmark, raising concerns about the reliability of AI model evaluations.

Jul 813